Imagine you're organizing a big tech conference, and you want to understand what people thought of your conference, so you can run it even better next year.
A good resource to look at, would be what people posted on Social Media about your conference, because well, it's a tech event after all. But there's a problem here: if your event is as popular as the Web Summit, you're going to have to go through hundreds of thousands of tweets, which is simply not practical.
One common solution would be to use Social Media monitoring and analysis tools, which try to aggregate all of these posts in one place, and crunch them into more understandable metrics: total number of posts, most frequently used hashtags, average sentiment, etc., and this is indeed what we did in our 2014 report of the Web Summit according to Twitter.
But what if we wanted to go a level deeper, to understand a little bit more in depth, what people thought and said about our conference? What if we wanted to build a map, that shows all the topics that were discussed, the sentiment around them, and how these two things evolved with time? That's what we're going to explore in this series of blogs.
Over a 5 day period, we collected about 77,000 tweets about the Web Summit 2015 using the Twitter Streaming API and in this blog series, we're going to explore them and see what we can extract.
Note: This is a step by step guide which contains all you need to accomplish (almost) the same results, so feel free to try it for yourself.
In the first part of this series, we're going to build a 2D map of the tweets that we've collected, to get a better sense of what people talked about at a high level. We're going to try and put the similar tweets closer together on this map, and the dissimilar ones further apart.
We'll try out a few different methods such as classical document vectors with tf-idf scoring, K-means clustering, Latent Dirichlet Allocation (LDA), averaged Word Vectors (using GloVe word embeddings), Paragraph Vectors (using gensim's doc2vec implementation) and finally Skip-Thought Vectors to group together similar tweets, and we will then use a dimensionality reduction algorithm called t-SNE to give each tweet an (x,y) co-ordinate in a 2D space, that allows us to put the tweets on a scatter plot.
In [1]:
import json
tweets_file = '/Users/parsa/Desktop/WebSummit 2015/websummit_dump_20151106155110'
with open(tweets_file) as f:
tweets = json.load(f)
print('# of tweets:', len(tweets))
for tweet in tweets[:5]:
print(tweet["text"])
Next we're going to do some pre-processing to remove stopwords such as "the" and "a", convert the tweets to lower case, and normalize the URLs, user names, smiley faces, hashtags, elongated words, and numbers. We define the pre-processor class as follows:
In [2]:
import re
class TweetPreprocessor(object):
def __init__(self):
self.FLAGS = re.MULTILINE | re.DOTALL
self.ALLCAPS = '<allcaps>'
self.HASHTAG = '<hashtag>'
self.URL = '<url>'
self.USER = '<user>'
self.SMILE = '<smile>'
self.LOLFACE = '<lolface>'
self.SADFACE = '<sadface>'
self.NEUTRALFACE = '<neutralface>'
self.HEART = '<heart>'
self.NUMBER = '<number>'
self.REPEAT = '<repeat>'
self.ELONG = '<elong>'
def _hashtag(self, text):
text = text.group()
hashtag_body = text[1:]
if hashtag_body.isupper():
result = (self.HASHTAG + " {} " + self.ALLCAPS).format(hashtag_body)
else:
result = " ".join([self.HASHTAG] + re.split(r"(?=[A-Z])", hashtag_body, flags=self.FLAGS))
return result
def _allcaps(self, text):
text = text.group()
return text.lower() + ' ' + self.ALLCAPS
def preprocess(self, text):
eyes, nose = r"[8:=;]", r"['`\-]?"
re_sub = lambda pattern, repl: re.sub(pattern, repl, text, flags=self.FLAGS)
text = re_sub(r"https?:\/\/\S+\b|www\.(\w+\.)+\S*", self.URL)
text = re_sub(r"/"," / ")
text = re_sub(r"@\w+", self.USER)
text = re_sub(r"{}{}[)dD]+|[)dD]+{}{}".format(eyes, nose, nose, eyes), self.SMILE)
text = re_sub(r"{}{}p+".format(eyes, nose), self.LOLFACE)
text = re_sub(r"{}{}\(+|\)+{}{}".format(eyes, nose, nose, eyes), self.SADFACE)
text = re_sub(r"{}{}[\/|l*]".format(eyes, nose), self.NEUTRALFACE)
text = re_sub(r"<3", self.HEART)
text = re_sub(r"[-+]?[.\d]*[\d]+[:,.\d]*", self.NUMBER)
text = re_sub(r"#\S+", self._hashtag)
text = re_sub(r"([!?.]){2,}", r"\1 " + self.REPEAT)
text = re_sub(r"\b(\S*?)(.)\2{2,}\b", r"\1\2 " + self.ELONG)
text = re_sub(r"([A-Z]){2,}", self._allcaps)
return text.lower()
Let's create an instance of the pre-processor, and see how it works with an example:
In [3]:
tweet_processor = TweetPreprocessor()
# an example:
tweet = "@sarahtavel What #MustHave #tech gadget can you not travel without? Stop by stand D131 on Wed at #WebSummit #Dublin https://t.co/2wFLAVpGiV"
print("Before: " + tweet + "\n")
print("After: " + tweet_processor.preprocess(tweet))
We remove common words such as "the", "a", "an", "this", etc. using NLTK's list of English stopwords, and add a few extra items to remove tokens that don't contribute much to the meaning of a tweet.
We also iterate over the list of tweets and add a new attribute processed to each tweet, which contains the processed and cleaned text.
In [4]:
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import TweetTokenizer
tknzr = TweetTokenizer()
nltk.download('stopwords')
stop = stopwords.words('english')
stop += ['<hashtag>', '<url>', '<allcaps>', '<number>', '<user>', '<repeat>', '<elong>', 'websummit']
for tweet in tweets:
parts = tknzr.tokenize(tweet_processor.preprocess(tweet["text"]))
clean = [i for i in parts if i not in stop]
tweet["processed"] = clean
And here's the end result of our pre-processing pipeline:
In [5]:
print("Before: " + tweets[1]["text"] + "\n")
print("After: " + str(tweets[1]["processed"]))
In [6]:
from sklearn.feature_extraction.text import TfidfVectorizer
tweet_texts = [tweet["text"] for tweet in tweets] # list of all tweet texts
tweet_texts_processed = [str.join(" ", tweet["processed"]) for tweet in tweets] # list of pre-processed tweet texts
vectorizer = TfidfVectorizer(min_df=4, max_features = 10000)
vz = vectorizer.fit_transform(tweet_texts_processed)
tfidf = dict(zip(vectorizer.get_feature_names(), vectorizer.idf_))
Let's check the tf-idf scores for a few random words:
In [7]:
print("dublin: " + str(tfidf["dublin"]))
print("catmull: " + str(tfidf["catmull"]))
print("cosgrave: " + str(tfidf["cosgrave"]))
print("enda: " + str(tfidf["enda"]))
print("sheeps: " + str(tfidf["sheeps"]))
We're going to create our first map of tweets, by plotting them on a 2D plane, based on pair-wise similarity between their corresponding document vectors. But as you might remember, we have created 10,000 features for each document, and therefore would need a 10,000-dimensional space to visualize their corresponding vectors!
That doesn't sound very practical or intuitive, so we have to reduce the dimensionality of these vectors first: from 10,000 features down to 2 or 3, depending if we want to visualize them in a 2- or 3-dimensional space.
Two common dimensionality reduction techniques are principal component analysis (PCA) and singular value decomposition (SVD). Here, we are going to use SVD to reduce the dimensionality to 50 dimensions first, and then use another dimensionality reduction technique called t-SNE that is particularly suited to visualizing high-dimensional datasets, to further reduce the dimensionality to 2.
In order to still be able to inspect individual points in the visualization, we visualize only part of our data: the first 10,000 tweets. Applying TruncatedSVD on this 10,000 x 10,000 matrix reduces it to the number of components specified and thus yields a 10,000 x 50 dimensional matrix that we then feed into t-SNE to produce a 10,000 x 2 matrix that we're able to visualize.
In [8]:
from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(n_components=50, random_state=0)
svd_tfidf = svd.fit_transform(vz[:10000])
In [9]:
svd_tfidf.shape
Out[9]:
In [10]:
from sklearn.manifold import TSNE
tsne_model = TSNE(n_components=2, verbose=1, random_state=0)
tsne_tfidf = tsne_model.fit_transform(svd_tfidf)
In [11]:
tsne_tfidf.shape
Out[11]:
In [12]:
tsne_tfidf[0]
Out[12]:
We are left with 10,000 (x,y) coordinates that we're going to visualize as a scatter plot. We also create tooltips to show the actual and processed text of each tweet (hover your mouse over the blue dots to see this).
In [13]:
import bokeh.plotting as bp
from bokeh.models import HoverTool, BoxSelectTool
from bokeh.plotting import figure, show, output_notebook
output_notebook()
plot_tfidf = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (tf-idf)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_tfidf.scatter(x=tsne_tfidf[:,0], y=tsne_tfidf[:,1],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000]
}))
hover = plot_tfidf.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\")"}
show(plot_tfidf)
Not bad for a first attempt: the tweets are separated nicely, and have formed a couple of homogenous 'islands'. For instance, there's an island of tweets about "pub crawls" on the top left, and one about the "future of Ireland" on the right hand side. But there are two fundamental problems with this chart:
K-means is a popular clustering algorithm that tries to distribute a predefined number of points (K) in a way that they end up in the center of our clusters, close to the mean.
We're going to create 10 clusters using MiniBatchKMeans from scikit-learn, which is a fast implementation of k-means that processes examples in small batches instead of individually.
In [14]:
from sklearn.cluster import MiniBatchKMeans
num_clusters = 10
kmeans_model = MiniBatchKMeans(n_clusters=num_clusters, init='k-means++', n_init=1,
init_size=1000, batch_size=1000, verbose=False, max_iter=1000)
kmeans = kmeans_model.fit(vz)
kmeans_clusters = kmeans.predict(vz)
kmeans_distances = kmeans.transform(vz)
The k-means algorithm runs for a few hundred iterations until the centroids don't improve much any more, and then for each tweet, it provides us with the closest centroid and the distance to each cluster centroid.
Let's see which cluster the first five tweets – that we saw earlier – have ended up in:
In [15]:
for i, tweet in enumerate(tweets):
if(i < 5):
print("Cluster " + str(kmeans_clusters[i]) + ": " + tweet["text"] + "(distance: " + str(kmeans_distances[i][kmeans_clusters[i]]) + ")")
To better understand what's in each cluster, let's get the top 10 features (word) for each of our 10 clusters:
In [16]:
sorted_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(num_clusters):
print("Cluster %d:" % i, end='')
for j in sorted_centroids[i, :10]:
print(' %s' % terms[j], end='')
print()
As you can see there's some meaningful consolidation in some of these clusters:
Now let's try and visualize the tweets again, according to their distance from each centroid in our K clusters. We are going to use t-SNE again to reduce the dimensionality from 10 (we had 10 clusters/centroids) down to 2.
In [17]:
tsne_kmeans = tsne_model.fit_transform(kmeans_distances[:10000])
To make the separation stand out more clearly, in addition to plotting the tweets, we are going to colorize each tweet according to the cluster it belongs to:
In [18]:
import numpy as np
colormap = np.array([
"#1f77b4", "#aec7e8", "#ff7f0e", "#ffbb78", "#2ca02c",
"#98df8a", "#d62728", "#ff9896", "#9467bd", "#c5b0d5",
"#8c564b", "#c49c94", "#e377c2", "#f7b6d2", "#7f7f7f",
"#c7c7c7", "#bcbd22", "#dbdb8d", "#17becf", "#9edae5"
])
plot_kmeans = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (k-means)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_kmeans.scatter(x=tsne_kmeans[:,0], y=tsne_kmeans[:,1],
color=colormap[kmeans_clusters][:10000],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000],
"cluster": kmeans_clusters[:10000]
}))
hover = plot_kmeans.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\" - cluster: @cluster)"}
show(plot_kmeans)
The separation seems to have improved, but there are some overlaps between different clusters, and the islands are still formed around keywords and keyphrases.
Next we're going to use a well-known topic modeling algorithm called LDA, to uncover the latent topics in the tweets, and then wet're going to use the topic distributions for each document as a measure to group similar tweets together.
First we vectorize our data by representing each tweet as a 10k dimensional vector whose indices correspond to the 10k most frequent terms in our corpus. We then feed this 77,000 x 10,000 feature matrix into LDA to detect the latent topics in our data. While there are non-parametric alternatives available, the classic version of LDA allows us to specify the number of topics that we want to discover upfront. We run LDA for 2,000 iterations to identify 15 topics and receive a 77k x 15 matrix of topic distributions of our tweets.
In [19]:
import lda
from sklearn.feature_extraction.text import CountVectorizer
cvectorizer = CountVectorizer(min_df=4, max_features=10000, stop_words='english')
cvz = cvectorizer.fit_transform(tweet_texts_processed)
n_topics = 15
n_iter = 2000
lda_model = lda.LDA(n_topics=n_topics, n_iter=n_iter)
X_topics = lda_model.fit_transform(cvz)
Let us now look at those topics in more detail. Specifically, we can inspect the words that are most relevant to a topic. We save these words as topic summaries for later.
In [20]:
n_top_words = 8
topic_summaries = []
topic_word = lda_model.topic_word_ # get the topic words
vocab = cvectorizer.get_feature_names()
for i, topic_dist in enumerate(topic_word):
topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
topic_summaries.append(' '.join(topic_words))
print('Topic {}: {}'.format(i, ' '.join(topic_words)))
Most of these topics look somewhat coherent:
To visualize the tweets according to their topic distributions, we first need to reduce the dimensionality from 15 down to 2 using t-SNE:
In [21]:
tsne_lda = tsne_model.fit_transform(X_topics[:10000])
Let's get the main topic for each tweet, which we'll use to colorize them later on:
In [22]:
doc_topic = lda_model.doc_topic_
lda_keys = []
for i, tweet in enumerate(tweets):
lda_keys += [doc_topic[i].argmax()]
In [23]:
plot_lda = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (LDA)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_lda.scatter(x=tsne_lda[:,0], y=tsne_lda[:,1],
color=colormap[lda_keys][:10000],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000],
"topic_key": lda_keys[:10000]
}))
hover = plot_lda.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\" - topic: @topic_key)"}
show(plot_lda)
This seems to be our best chart so far: there are entire islands dedicated to marketing or partying, without those words getting explicitly mentioned, so we've took a solid step towards a concept or topic-based representation.
Moreover, the chart not only puts similar tweets together, but also brings similar topics closer. Pay attention to how close business (grey) and marketing (dark pink) are, and how far innovation/future (orange) is pushed away from fun and partying (purple).
Word Embeddings are everywhere these days, and as the flagship of Deep Learning-based techniques for processing language, they have dominated the world of NLP research since 2014, and for good reasons.
In layperson terms, these are dense vectors of typically between 25 and 300 dimensions that capture a lot of information about the word as well as its surrounding context. In this respect, they can be seen as another form of dimensionality reduction (such as PCA, SVD, LDA, etc.) that has been shown to be very effective in a wide array of NLP tasks.
These vectors have a set of characteristics that make them very special. To begin with, the vectors of similar words are closer to each other, and conversely, the vectors of dissimilar words are far from each other.
Moreover, the vectors can be combined to find answers to queries such as "king is to man as queen is to ???" by forming simple arithmetic equations such as:
$$v_{king} - v_{man} ~= v_{queen} - v_{woman}$$Let's see if we can exploit these characteristics to enhance our map.
Our first step is to read the pretrained GloVe vectors into a list:
In [24]:
print("loading glove model...")
embedding_size = 50
glove_file = '/Users/parsa/Downloads/glove.twitter.27B/glove.twitter.27B.50d.txt'
glove = {}
with open(glove_file) as f:
for line in f.readlines():
line = line.replace("\n","").split(" ")
glove[line[0]] = np.array(line[1:],dtype='float64')
The model is loaded. Let's look at the vector for the word "book":
In [25]:
print("book:", glove["book"])
In order to get a vector representation for an entire tweet, we're going to combine the vectors of individual words in it, by doing a weighted average using tf-idf values ($t$), to give more importance to vectors of more significant words:
$$v_{tweet} = \frac{1}{|T|}\sum_{i \in T}{v_i.t_i}$$
In [26]:
def tweetVector(processed_tweet):
l = float(len(processed_tweet) | 1)
sum = np.zeros(embedding_size)
for part in processed_tweet:
sum += glove.get(part, np.zeros(embedding_size)) * tfidf.get(part, 1)
return sum/l
The tweetVector function takes a tweet, and returns a 50-dimensional vector by averaging the individual word vectors. Next we're going to run through our tweets, and calculate the tweet vector for each one of them:
In [27]:
for tweet in tweets:
tweet["vector"] = tweetVector(tweet["processed"])
Let's take a look at the vector for our first tweet:
In [28]:
tweets[0]["vector"]
Out[28]:
Note that a caveat of this approach is that we lose word order, and in that sense we're doing something similar to a bag-of-words model. For instance, the following sentences have the same tweet vectors:
In [29]:
np.allclose(tweetVector(["we", "are", "totally", "equal"]), tweetVector(["equal", "totally", "we", "are"]))
Out[29]:
Now that we've calculated the tweet vectors, let's release the GloVe embeddings and free up some memory:
In [30]:
glove = None
In [31]:
sample_size = 10000
filtered = [tweet for tweet in tweets if "<url>" not in tweet["processed"]]
sorted_ = sorted(filtered, key=lambda tweet: len(tweet["text"]), reverse=True)
tweets_sample = sorted_[:sample_size]
The tweet vectors that we've created are 50-dimensional, which means we now have a matrix of 77,000 x 50 that we need to plot. We're going to use t-SNE again to reduce the dimensionality down to 2:
In [32]:
tsne_glove = tsne_model.fit_transform([tweet["vector"] for tweet in tweets[:10000]])
Time to plot the tweet vectors on a 2D surface. Note that since we were happy with the LDA topics, we're going to use the topic keys again to colorize the tweets, and check if they correlate with the spatial separation:
In [33]:
plot_glove = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (averaged GloVe vectors)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_glove.scatter(x=tsne_glove[:,0], y=tsne_glove[:,1],
color=colormap[lda_keys][:10000],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000]
}))
hover = plot_glove.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\")"}
show(plot_glove)
No clear separation is evident in this representation, though there are areas that are clearly richer in one or two colors. The word vectors have worked their magic though, and neighboring tweets are a lot closer to each other semantically, without necessarily sharing the same keywords.
Paragraph Vector is an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. The algorithm represents each document by a dense vector which is trained to predict words in the document.
We are going to use the implementation available in gensim, called doc2vec, to created paragraph vectors for each tweet in our dataset:
In [34]:
import gensim
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
Let's extract the documents and tokenize them (without removing the stopwords):
In [35]:
docs = [TaggedDocument(tknzr.tokenize(tweet_processor.preprocess(tweet["text"])), [i]) for i, tweet in enumerate(tweets)]
Here's what a tokenized document looks like:
In [36]:
docs[0]
Out[36]:
We train our doc2vec model with 100 dimensions, a window of size 8, a minimum word count of 5 and with 4 workers:
In [37]:
doc2vec_model = Doc2Vec(docs, size=100, window=8, min_count=5, workers=4)
We can do some fun things with the trained model. Let's see if we can find out who's Dell's CEO, by knowing Pixar's CEO:
In [38]:
doc2vec_model.most_similar(positive=["pixar", "dell"], negative=["catmull"])
Out[38]:
The second entry is "michael" which refers to Michael Dell, CEO of Dell. Note that the model picked this up from the 77k tweets alone, with no additional knowledge source.
Let' see what each paragraph vector looks like:
In [39]:
doc2vec_model.docvecs[0]
Out[39]:
Now we're going to use t-SNE to reduce dimensionality and plot the tweets:
In [40]:
doc_vectors = [doc2vec_model.docvecs[i] for i, t in enumerate(tweets[:10000])]
In [41]:
tsne_d2v = tsne_model.fit_transform(doc_vectors)
Let's put the tweets on a 2D plane:
In [42]:
plot_d2v = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (doc2vec)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_d2v.scatter(x=tsne_d2v[:,0], y=tsne_d2v[:,1],
color=colormap[lda_keys][:10000],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000]
}))
hover = plot_d2v.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\")"}
show(plot_d2v)
This is a bit disappointing both in terms of separation, as well as semantic similarity of neighboring tweets. Perhaps it can be improved by plugging in word embeddings trained on an external corpus such as Wikipedia. This can be done using the intersect_word2vec_format method in the model object but we're not going to do that here.
Another Deep Learning-based representational model that has gained popularity recently is Skip-Thought Vectors, also called Sent2Vec. It's described as an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, an encoder-decoder model is trained, that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations.
We have used the implementation and models made available by the author, to create Skip-Thought Vectors for our tweets:
In [43]:
import pickle
skipthought_vectors = pickle.load(open("/Users/parsa/Desktop/WebSummit 2015/skip-thought/skip-thoughts/tweet_skipthought_vectors.pk", "rb"), encoding='latin1')
In [44]:
skipthought_vectors.shape
Out[44]:
We're now going to use t-SNE to reduce the dimensionality from 4800 to 2:
In [45]:
tsne_st = tsne_model.fit_transform(skipthought_vectors[:10000])
Let's plot these vectors:
In [46]:
plot_st = bp.figure(plot_width=900, plot_height=700, title="Web Summit 2015 tweets (Skip-Thought Vectors)",
tools="pan,wheel_zoom,box_zoom,reset,hover,previewsave",
x_axis_type=None, y_axis_type=None, min_border=1)
plot_st.scatter(x=tsne_st[:,0], y=tsne_st[:,1],
color=colormap[lda_keys][:10000],
source=bp.ColumnDataSource({
"tweet": tweet_texts[:10000],
"processed": tweet_texts_processed[:10000]
}))
hover = plot_st.select(dict(type=HoverTool))
hover.tooltips={"tweet": "@tweet (processed: \"@processed\")"}
show(plot_st)
There are some nice consolidations here, such as one where most tweets are about excitement and anticipation, but we couldn't find a general pattern for topical similarity. Perhaps this is due to the difference between the two contexts: the model is trained on books, whereas our documents are tweets, and may include words and phrases that don't exist in our model's vocabulary.
We tried several different methods to build a semantic map of our 77k tweets. Each method had its own weaknesses and advantages, but the LDA one looks the most intuitive to us so far.
Here are a few things to try next:
In our next blog in the series we're going to explore in more detail, how the topics evolved over the course of the event, along with the sentiment towards each topic.
Special thanks to:
We got inspiration for this report from: